Static and dynamic overproduction and selection of classifier ensembles with genetic algorithms

The overproduce-and-choose sttategy is a static classifier ensemble selection approach, which is divided into overproduction and selection phases. This thesis focuses on the selection phase, which is the challenge in overproduce-and-choose strategy. When this phase is implemented as an optimization process, the search criterion and the search algorithm are the two major topics involved. In this thesis, we concentrate in optimization processes conducted using genetic algorithms guided by both single- and multi-objective functions. We first focus on finding the best search criterion. Various search criteria are investigated, such as diversity, the error rate and ensemble size. Error rate and diversity measures are directly compared in the single-objective optimization approach. Diversity measures are combined with the error rate and with ensemble size, in pairs of objective functions, to guide the multi-optimization approach. Experimental results are presented and discussed. Thereafter, we show that besides focusing on the characteristics of the decision profiles of ensemble members, the control of overfitting at the selection phase of overproduce-and-choose strategy must also be taken into account. We show how overfitting can be detected at the selection phase and present three strategies to control overfitting. These strategies are tailored for the classifier ensemble selection problcm and compared. This comparison allows us to show that a global validation strategy should be applied to control overfitting in optimization processes involving a classifier ensembles selection task. Furthermore, this study has helped us establish that this global validation strategy can be used as a tool to measure the relationship between diversity and classification performance when diversity measures are employed as single-objective functions. Finally, the main contribution of this thesis is a proposed dynamic overproduce-and-choose strategy. While the static overproduce-and-choose selection strategy has traditionally focused on finding the most accurate subset of classifiers during the selection phase, and using it to predict the class of all the test samples, our dynamic overproduce-and- choose strategy allows the selection of the most confident subset of classifiers to label each test sample individually. Our method combines optimization and dynamic selection in a two-level selection phase. The optimization level is intended to generate a population of highly accurate classifier ensembles, while the dynamic selection level applies measures of confidence in order to select the ensemble with the highest degree of confidence in the current decision. Three different confidence measures are presented and compared. Our method outperforms classical static and dynamic selection strategies

Theory and application of support vector machines to learning and recognition of objects based on appearance.

Support Vector Machines (SVM) é uma técnica de aprendizagem de máquina derivada de duas fundamentações sólidas: Teoria da Aprendizagem Estatísta e Otimização Matemática. SVM têm sido recentemente aplicado com sucesso a uma variedade de problemas que vão desde o reconhecimento de caracteres ao reconhecimento de objetos baseado na aparência. Alguns dos motivos para esse sucesso estão relacionados ao fato dessa técnica exibir bom desempenho de generalização em muitas bases de dados reais, é bem fundamentada teóricamente, o processo de treinamento elimina a possibilidade de mínimos locais, existem poucos parâmetros livres para ajustar e a arquitetura não precisa ser encontrada por experimentação. Entretanto, por tratar-se de uma abordagem relativamente nova, livros-texto e artigos estão geralmente disponíveis em uma linguagem que não é facilmente acessível para Cientistas da Computação. Portanto, um dos objetivos desta dissertação é prover uma introdução sobre SVM que apresente os conceitos e teoria essenciais à técnica e que seja mais didática. Estratégias de reconhecimento de objetos com base na aparência se aplicam a problemas em que há dificuldades na obtenção de modelos geométricos dos objetos, desde que as imagens utilizadas não apresentem oclusões. Algumas técnicas de aprendizagem de máquina têm sido aplicadas a este problema, tais como: PCA (Principal Component Analysis), PAC (Probably Approximately Correct) e Redes Neurais, mas nenhuma mostrou-se tão promissora quanto SVM. Dentro desse contexto, esta dissertação objetiva investigar a aplicação de SVM ao reconhecimento de objetos baseado na aparência. Apresenta resultados práticos de classificação utilizando inicialmente uma pequena base de dados e, em seguida, explorando todo o poder da técnica em uma base de dados relativamente grande. Esta dissertação também descreve resultados experimentais usando diferentes variações da técnica e compara o desempenho de reconhecimento de SVM com o desempenho de Redes Neurais do tipo Multilayer Perceptron Backpropagation.Support Vector Machines (SVM) is a machine learning technique derived from two solid backgrounds: Statistical Learning Theory and Mathematical Optimisation. SVM has recently been applied with success to a variety of problems, ranging from character recognition to appearance based object recognition. Some of the reasons for this success are related to the fact this technique exhibits good generalisation performance on many real-life data sets, is well-founded theoreticaly, the training process eliminates the possibility of local minima, there are few free paramets to adjust and the architecture does not have to be found by experimentation. However, since this is a relatively new approach, text books and papers are usually in a language that is not easily acessible to Computer Scientists. Therefore one of the objectives of this dissertation is to provide an introduction to SVM that presents the essential concepts and theory behind the technique and that is more didatic. Appearance-based object recognition strategies appear to be well-suited for the solution of recognition problems in which geometric models of the viewed objects can be difficult to obtain, although they are not naturally tolerant to occlusions. Some machine learning techniques have been applied in this problem like, Principal Component Analysis (PCA), Probably Approximately Correct (PAC) and Neural Networks, but none posed as promising as SVM. Within this context, this dissertation aims to investigate the application of SVM to appearance-based object recognition. It presents practical results of classification initially using a small dataset and then exploring the full power of the technique on a relatively large dataset. It also presents experimental results using different variations of the technique and compares the recognition performance of SVM with the performance of Multilayer Percep tron Backpropagation Neural Networks